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DETAILED ACTION 

1 . This action is responsive to Application filed 1 1/03/2003 . 

2. Claims 1-20 are currently pending in this application. Claims 1, 7, 12 and 16 are 
independent claims. 

Claim Rejections - 35 USC §101 

3. 35U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title, 

4. Claims 1-5 and 16-20 are rejected under 35 U S.C. 101 because the claimed invention is 
directed to non-statutory subject matter for the reason set forth below: 

Regarding independent claims 1-5 and 16-20 are not limited to tangible embodiments. 
Claims 1-5 and 16-20 discloses the process such as, "A method, comprising: receiving...storing, 
modifying...outputting, detecting..." associating with the concept of annotating of media 
information, are interpreted as software per se, abstracts ideas or mental construct and not 
tangibly embodied on a computer readable medium or hardware (see WHAT IS THE 
CLAIMED: pages 14 and 12-18), set forth descriptive material but fail to set forth physical 
structures or materials comprising of hardware or a combination of hardware and software to 
produce a "useful and tangible" result; 

(1) "USEFUL RESULT", For an invention to be "useful" it must satisfy the utility 
requirement of section 101 see MPEP § 2107 and Fisher, 421 F.3d at , 76 USPQ2d at 1230 
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(citing the Utility Guidelines with approval for interpretation of "specific" and "substantial"), 
and 

(2) "TANGIBLE RESULT", For an invention to be "useful" it must satisfy the utility 
requirement of section 101 see MPEP § 2107 and see Benson, 409 U.S. at 71-72, 175 USPQ at 
676-77 (invention ineligible because had "no substantial practical application "). "[A]n 
application of a law of nature or mathematical formula to a . . . process may well be deserving of 
patent protection " Diehr, 450 US. at 187, 209 USPQ at 8 (emphasis added); see also Corning, 
56 U.S. (15 How.) at 268, 14 L.Ed. 683 ("It is for the discovery or invention of some practical 
method or means of producing a beneficial result or effect, that a patent is granted . . "). 

Claim Rejections - 35 USC §103 

5. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

6. Claims 1-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Nelson et al. 
US 20040236830A1 filed 05/15/2003 (hereinafter Nelson), in view of King et al. 
US005600775A issued 02/04/1997 (hereinafter King). 

In regard to independent claim 1, receiving an indication that annotation of media 
information is desired; storing annotation information (see Nelson at page 1 paragraphs 
[0006] through page 2 paragraph [0014]), describes a systems that provides the ability to be 
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notified of annotations through electronic mail (email). Furthermore, once the annotations are 
made, there is no mechanism for reconstructing the annotations for future reference. Thus, if a 
person misses the videoconference session for whatever reason, the data is lost; to overcome the 
above coming the above, Nelson' 830 discloses an annotation management configured to mange 
and store annotation data annotation control data and also configured to provide real-time 
annotation daja to clients of a videoconference session is provided. The method initiates with 
annotating a display region of a user interface associated with a client of the videoconference 
session. Then, annotating of the display region is detected, 

and modifying ... of the media information to reflect a presence of the annotation 
information (see Nelson at page 1 paragraph [0001 1] through page 2 paragraph [0014]), 
discloses the storage server is configured to store the media data and the annotation data, wherein 
an event database in communication with the media management server is included to capture 
events associated with the annotation data. The media analysis server is configured to associate 
the stored annotation data with the captured events to enable reconstruction of the 
videoconference session based on the captured events and a graphical user interface (GUI) 
enabled to provide real-time annotation of display data rendered on a display screen is provided. 
The GUI includes a media display region corresponding to a media signal. The media display 
region is capable of being annotated by a videoconference participant, wherein the annotation of 
the media display region generates an event for storage on an annotation management server. 
The annotation of the media display region further generates a signal presented to remaining 
videoconference participants in real-time. A control display region enabling a participant to 
define control properties associated with the media display region is included. 
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Nelson does not explicitly teach, and modifying an index of the media information to 
reflect a presence of the annotation information, however (see King at col. 2, line lthrough 
col. 3, line 30, also see Fig. 1), discloses the video and annotations are stored separately that 
provides an indexing scheme relates the annotations to the video frames. On full motion video 
playback, the annotations are displayed on the originally annotated frames. The annotations can 
be manipulated independently of the video information. For example, the annotations and video 
can be transported separately, while the indexing scheme allows the annotations and the video to 
be recombined at the receiving station. 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to have modified Nelson's teaching, discloses an annotation management 
configured to receiving an indication that annotation of media information is desired; storing 
annotation information and modifying ... of the media information to reflect a presence of the 
annotation information, to includes a means of modifying an index of the media information to 
reflect a presence of the annotation information, of King's teaching. One of ordinary skill in the 
art would have been motivated to modify this combination, because they are from the same field 
of endeavor of media annotation control and/or management and desirable to provide an 
annotation tool that allows reductions in production time, more effective utilization of resources, 
and greater communication in general during creation of multimedia products or other large scale 
indexed data files in real time interaction among developers and /or users (see King at col. 1 lines 
25-67). 
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In regard to independent claim 7, incorporates substantially similar subject matter as 
cited in claim 1 above, and further view of the following, and is similarly rejected along the same 
rationale, 

an interface ...a memory, media information, ...a processor (see Nelson at page 1 
paragraph [0008] through page 6 paragraph [0055]), discloses a process, a system, a computer 
readable media, or a graphical user interface, wherein the annotation data is preserved so that the 
videoconference session may be reconstructed using computer readable code on a computer 
readable medium. The computer readable medium is any data storage device that can store data, 
which can be thereafter read by a computer system. The computer readable medium also 
includes an electromagnetic carrier wave in which the computer code is embodied. Examples of 
the computer readable medium include hard drives, network attached storage (NAS), read-only 
memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other 
optical and non-optical data storage devices. The computer readable medium can also be 
distributed over a network coupled computer system so that the computer readable code is stored 
and executed in a distributed fashion), 

and to selectively retrieve the annotation information from the memory based on the 
index information; and a display module to combine the media information and the 
annotation information for output to a display device, however (see King at col. 2, line 1 
through col. 5, line 25, also see Fig. 1), discloses the video and annotations are stored separately 
that provides an indexing scheme relates the annotations to the video frames. On full motion 
video playback, the annotations are displayed on the originally annotated frames. The 
annotations can be manipulated independently of the video information. For example, the 
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annotations and video can be transported separately, while the indexing scheme allows the 
annotations and the video to be recombined at the receiving station. Furthermore, provides input 
tools in the instruction memory and possibly other input devices in order to store input data into 
the annotation data structures in the data memory. A graphical element is associated with each 
annotated data structure. This graphic element is displayed as an overlaying annotation in the 
video window, where the annotation overlies the video frame, without modifying the video 
frame during playback of an annotated file. 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to have modified Nelson's teaching, discloses an annotation management 
configured to receiving an indication that annotation of media information is desired; storing 
annotation information and modifying ... of the media information to reflect a presence of the 
annotation information, to includes a means of selectively retrieve the annotation information 
from the memory based on the index information; and a display module to combine the media 
information and the annotation information for output to a display device, of King's teaching. 
One of ordinary skill in the art would have been motivated to modify this combination, because 
they are from the same field of endeavor of media annotation control and/or management and 
desirable to provide an annotation tool that allows reductions in production time, more effective 
utilization of resources, and greater communication in general during creation of multimedia 
products or other large scale indexed data files in real time interaction among developers and /or 
users (see King at col. 1 lines 25-67). 
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In regard to independent claim 12, incorporates substantially similar subject matter as 
cited in claims 1 and 7 above, and further view of the following, and is similarly rejected along 
the same rationale, 

Examiner read an article of manufacture in the broadest reasonable interpretation to the 
claim limitation; which would have been an obvious variant of media information of claims 1 
and 7 above, to a person of ordinary skill in the art at the time the invention was made, and 

detecting an annotation marker in the index file; retrieving annotation information 
associated with the annotation marker; and combining the media information and the 
annotation information to display annotated media information, however (see King at the 
Abstract and col. 2, line 1 through col. 6, line 65, also see Fig. 1 and 3), discloses the video and 
annotations are stored separately that provides an indexing scheme relates the annotations to the 
video frames. During playback, the annotation memory is monitored to detect annotation data 
structure for indexed data structures currently being displayed. If an annotation data structure is 
detected for the current indexed data structure, a graphical element overlies the indexed data 
structure on the display without modifying the indexed data structure. Text process documents 
can be handled much in the same manner. Also, annotation from several users may be merged 
and viewed in one batch. Furthermore (see King in detail at col. 6, lines 5-65), discloses the 
detail of the indexing schema, such as annotation indication of the index, each multimedia 
indication of the index and so on, so that on full motion video playback, the annotations are 
displayed on the originally annotated frames. The annotations can be manipulated independently 
of the video information. For example, the annotations and video can be transported separately, 
while the indexing scheme allows the annotations and the video to be recombined at the 
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receiving station. Furthermore, provides input tools in the instruction memory and possibly other 
input devices in order to store input data into the annotation data structures in the data memory. 
A graphical element is associated with each annotated data structure. This graphic element is 
displayed as an overlaying annotation in the video window, where the annotation overlies the 
video frame, without modifying the video frame during playback of an annotated file, 

Examiner read the above in the broadest reasonable interpretation to the claim limitation; 
wherein the annotation marker would have been an obvious variant of annotation indication of 
the index, to a person of ordinary skill in the art at the time the invention was made. 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to have modified Nelson's teaching, discloses an annotation management 
configured to receiving an indication that annotation of media information is desired; storing 
annotation information and modifying ... of the media information to reflect a presence of the 
annotation information, to includes a means of detecting an annotation marker in the index file; 
retrieving annotation information associated with the annotation marker; and combining the 
media information and the annotation information to display annotated media information, of 
King's teaching. One of ordinary skill in the art would have been motivated to modify this 
combination, because they are from the same field of endeavor of media annotation control 
and/or management and desirable to provide an annotation tool that allows reductions in 
production time, more effective utilization of resources, and greater communication in general 
during creation of multimedia products or other large scale indexed data files in real time 
interaction among developers and /or users (see King at col. 1 lines 25-67). 
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In regard to independent claim 16, incorporates substantially similar subject matter as 
cited in claims 1, 7 and 12 above, and further view of the following, and is similarly rejected 
along the same rationale, 

Examiner read the above in the broadest reasonable interpretation to the claim limitation; 
wherein a point in the index file would have been an obvious variant of marker in the index file 
of claim 12, thus incorporates substantially similar subject matter, to a person of ordinary skill in 
the art at the time the invention was made. 

In regard to dependent claim 2, querying a user for a source of the annotation 
information before the storing (see Nelson at page 2 paragraph [0013]), providing real-time 
annotation data to clients with annotating a display region of a user interface associated with a 
client of the videoconference session. Then, annotating of the display region is detected. In 
response to detecting the annotating of the display region, the method includes communicating 
data corresponding to the detecting of the annotating of the display region to other clients of the 
videoconference session for real-time presentation. Next, the data corresponding to the detecting 
of the annotating of the display region is stored. Then, the data corresponding to the detecting of 
the annotating of the display region is associated with data defining the videoconference 
session). 

In regard to dependent claim 3, incorporates substantially similar subject matter as 
cited in claims 1, and 7 above, and is similarly rejected along the same rationale. 

In regard to dependent claim 4, wherein the annotation information includes control 
data, text, audio information, graphical information, or video information (see Nelson at 
page 1 paragraph [0008] through page 6 paragraph [0055]), discloses a process, a system, a 
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computer readable media, or a graphical user interface, wherein the annotation data is preserved 
so that the videoconference session may be reconstructed using computer readable code on a 
computer readable medium. 

In regard to dependent claims 5-6, incorporates substantially similar subject matter as 
cited in claims 1, 7, 12 and 16 above, and is similarly rejected along the same rationale. 

In regard to dependent claim 8, a tuner connected to the processor to separate the 
media information from an input media stream, however (see King at col. 2, line 15 through 
col. 3, line 35, also see Fig. 1), discloses the video and annotations are stored separately that 
provides an indexing scheme relates the annotations to the video frames. Also can be 
characterized as an apparatus (see item 10 (cpu) of Fig. 1) for display and annotation of a file of 
indexed data structures, wherein types of annotations, which can be created, include free-hand 
bitmap drawings, text, and audio data. When audio data is used, an icon is used as the graphical 
element of the annotation data structure, which can be displayed over the indexed data structure 
to indicate presence of an audio annotation. Also, the annotation graphical element can be 
positioned on the screen overlaying the indexed data structure using a pointer device which 
inputs coordinate data for the annotation data structure. 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to have modified Nelson's teaching, discloses an annotation management 
configured to receiving an indication that annotation of media information is desired; storing 
annotation information and modifying ... of the media information to reflect a presence of the 
annotation information, to includes a tuner that connected to the processor to separate the media 
information from an input media stream, of King's teaching. One of ordinary skill in the art 
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would have been motivated to modify this combination, because they are from the same field of 
endeavor of media annotation control and/or management and desirable to provide an annotation 
tool that allows reductions in production time, more effective utilization of resources, and greater 
communication in general during creation of multimedia products or other large scale indexed 
data files in real time interaction among developers and /or users (see King at col 1 lines 25-67). 

In regard to dependent claims 9-10, incorporates substantially similar subject matter as 
cited in claims 3-4 and 7-8 above, and are similarly rejected along the same rationale. 

In regard to dependent claim 11, incorporates substantially similar subject matter as 
cited in claims 1 and 7 above, and further view of the following, and is similarly rejected along 
the same rationale, 

Examiner read a communication link to access annotation content referenced by the 
annotation information of claim 1 1 in the broadest reasonable interpretation to the claim 
limitation; which would have been an obvious variant of an interface to receive annotation 
information; a memory to store the annotation information, media information, and index 
information relating to the annotation information and the media information; a processor to 
retrieve the media information from the memory and to selectively retrieve the annotation 
information from the memory based on the index information of claims 1 and 7 above, to a 
person of ordinary skill in the art at the time the invention was made. 

In regard to dependent claim 13, incorporates substantially similar subject matter as 
cited in claims 2-3, 7, 12 and 16 and 7-8 above, and is similarly rejected along the same 
rationale. 
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In regard to dependent claim 14, incorporates substantially similar subject matter as 
cited in claims 1 and 7 above, and further view of the following, and is similarly rejected along 
the same rationale, 

overlaying the annotation information on the media information, however (see King 
at col. 2, line 15 through col. 3, line 35, also see Fig. 3), discloses the video and annotations are 
stored separately that provides an indexing scheme relates the annotations to the video frames. 

The annotation data structure includes a graphical element for display overlaying the 
selected data structure and an indication of an index value for the selected data structure. 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to have modified Nelson's teaching, discloses an annotation management 
configured to receiving an indication that annotation of media information is desired; storing 
annotation information and modifying ... of the media information to reflect a presence of the 
annotation information, to includes a means of overlaying the annotation information on the 
media information, of King's teaching. One of ordinary skill in the art would have been 
motivated to modify this combination, because they are from the same field of endeavor of 
media annotation control and/or management and desirable to provide an annotation tool that 
allows reductions in production time, more effective utilization of resources, and greater 
communication in general during creation of multimedia products or other large scale indexed 
data files in real time interaction among developers and /or users (see King at col. 1 lines 25-67). 

In regard to dependent claim 15, incorporates substantially similar subject matter as 
cited in claims 1, 7 and 12 above, and further view of the following, and is similarly rejected 
along the same rationale, 
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Examiner read blending the annotation information and the media information of claim 
15 in the broadest reasonable interpretation to the claim limitation; which would have been an 
obvious variant of modifying an index of the media information to reflect a presence of the 
annotation information of claims 1 and 7 above, to a person of ordinary skill in the art at the time 
the invention was made. 

In regard to dependent claim 17, asking for a type of the annotation information 
before the receiving and storing, however (see King at col. 2, line 15 through col. 3, line 35, 
also see Fig. 1), discloses the video and annotations are stored separately that provides an 
indexing scheme relates the annotations to the video frames, wherein types of annotations, which 
can be created, include free-hand bitmap drawings, text, and audio data. When audio data is 
used, an icon is used as the graphical element of the annotation data structure, which can be 
displayed over the indexed data structure to indicate presence of an audio annotation. Also, the 
annotation graphical element can be positioned on the screen overlaying the indexed data 
structure using a pointer device which inputs coordinate data for the annotation data structure. 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to have modified Nelson's teaching, discloses an annotation management 
configured to receiving an indication that annotation of media information is desired; storing 
annotation information and modifying ... of the media information to reflect a presence of the 
annotation information, to includes a means of asking for a type of the annotation information 
before the receiving and storing, of King's teaching. One of ordinary skill in the art would have 
been motivated to modify this combination, because they are from the same field of endeavor of 
media annotation control and/or management and desirable to provide an annotation tool that 



Application/Control Number: 10/700,910 Page 15 

Art Unit: 2176 

allows reductions in production time, more effective utilization of resources, and greater 
communication in general during creation of multimedia products or other large scale indexed 
data files in real time interaction among developers and /or users (see King at col. 1 lines 25-67). 

In regard to dependent claim 18, incorporates substantially similar subject matter as 
cited in claims 1, 7, 12 and 16 above, and further view of the following, and is similarly rejected 
along the same rationale. 

In regard to dependent claim 19, incorporates substantially similar subject matter as 
cited in claim 1 above, and further view of the following, and is similarly rejected along the same 
rationale. 

repeating the outputting stored media information based on an associated index fde 
before the detecting a reference to the stored annotation information (see Nelson at page 1 
paragraphs [0006] through page 2 paragraph [0014]), describes a systems that provides the 
ability to be notified of annotations through electronic mail (email). Furthermore, once the 
annotations are made, there is no mechanism for reconstructing the annotations for future 
reference. Thus, if a person misses the videoconference session for whatever reason, the data is 
lost; to overcome the above coming the above, Nelson'830 discloses an annotation management 
configured to mange and store annotation data annotation control data and also configured to 
provide real-time annotation data to clients of a videoconference session is provided. The method 
initiates with annotating a display region of a user interface associated with a client of the 
videoconference session. Then, annotating of the display region is detected, also (see Nelson at 
page 1 paragraph [0001 1] through page 2 paragraph [0014]), discloses the storage server is 
configured to store the media data and the annotation data, wherein an event database in 
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communication with the media management server is included to capture events associated with 
the annotation data. The media analysis server is configured to associate the stored annotation 
data with the captured events to enable reconstruction of the videoconference session based on 
the captured events and a graphical user interface (GUI) enabled to provide real-time annotation 
of display data rendered on a display screen is provided. The GUI includes a media display 
region corresponding to a media signal. The media display region is capable of being annotated 
by a videoconference participant, wherein the annotation of the media display region generates 
an event for storage on an annotation management server. The annotation of the media display 
region further generates a signal presented to remaining videoconference participants in real- 
time. A control display region enabling a participant to define control properties associated with 
the media display region is included. 

In regard to dependent claim 20, incorporates substantially similar subject matter as 
cited in claims 1, 7, 12 and 16 above, and further view of the following, and is similarly rejected 
along the same rationale, 

determining whether the annotation information should be displayed, and 
combining the media information and the annotation information if the determining 
determines that the annotation information should be displayed, however (see King at col. 
2, line 15 through col. 12, line 60), discloses the video and annotations are stored separately that 
provides an indexing scheme relates the annotations to the video frames, wherein types of 
annotations, which can be created, include free-hand bitmap drawings, text, and audio data. 
When audio data is used, an icon is used as the graphical element of the annotation data 
structure, which can be displayed over the indexed data structure to indicate presence of an audio 
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annotation. Also, the annotation graphical element can be positioned on the screen overlaying the 
indexed data structure using a pointer device which inputs coordinate data for the annotation data 
structure, 

further includes the output comparator (i.e. true, open or not or position within note?) 
(See table on col. 8), 

Examiner read the above in the broadest reasonable interpretation to the claim limitation; 
wherein whether the annotation information should be displayed would have been an obvious 
variant of further includes the output comparator (i.e. true, open or not or position within note?) 
(See table on col. 8), to a person of ordinary skill in the art at the time the invention was made. 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to have modified Nelson's teaching, discloses an annotation management 
configured to receiving an indication that annotation of media information is desired; storing 
annotation information and modifying ... of the media information to reflect a presence of the 
annotation information, to includes a means of determining whether the annotation information 
should be displayed, and combining the media information and the annotation information if the 
determining determines that the annotation information should be displayed, of King's teaching. 
One of ordinary skill in the art would have been motivated to modify this combination, because 
they are from the same field of endeavor of media annotation control and/or management and 
desirable to provide an annotation tool that allows reductions in production time, more effective 
utilization of resources, and greater communication in general during creation of multimedia 
products or other large scale indexed data files in real time interaction among developers and /or 
users (see King at col. 1 lines 25-67). 
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Conclusion 



7. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Quoc A. Tran whose telephone number is (571) 272-4103. The 
examiner can normally be reached on Monday through Friday from 9 AM to 5 PM EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Herndon R. Heather can be reached on (571) -272-4136. The fax phone number for 
the organization where this application or proceeding is assigned is (571)-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



Quoc A, Tran 
Patent Examiner 
Technology Center 2176 
March 15, 2006 




